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IMPROVEMENT-ORIENTED EVALUATION : AN LEA EXAMPLE 



INTRODUCTION 

Much criticism has been leveled against the Title I Evaluation and 
Reporting System (TIERS) because of its restricted scope and limited 
potential for use in program implementation and improvement (Barnes and 
Ginsburg, 1979; David, 1981; variderPlbeg* 1982; Holley f 1980). While 
this criticism is generally justified, it has never been the intention of 
the U.S. Education Department to restrict Title I (how Chapter 1) 
evaluation to the gathering of summative information aggregatable up to 
the federal level. Indeed, project staff are urged to complement TIERS 
by implementing additional evaluation activities to address local 
concerns (Wisler and Anderson, 1979), Sustained effects evaluation was 
subsequently promulgated to require local districts to conduct 
evaluations addressing local questions. 

In spite of these efforts to expand Title I/Chapter i evaluations and 
to make them more useful at the local level, small- and medium-sized 
school districts have tended to confine their evaluation activities to 
meeting TIERS requirements. Several factors have contributed to this 
state of affairs. Among them are: (a) federal urgings are often seen as 
ah encroachment of local prerogatives and are responded to with a minimum 
level of effort on the part of local districts; (b) most staff at a 
small- or medium-sized district have relatively little time to attend to 
evaluation activities because of other numerous responsibilities; and (c) 
budget cuts have made it necessary to curtail project activities 
including program evaluation. 
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These obstacles are, of course, not insurmountable. This paper 
describes evaluation activities conducted by a medium-sized district 
which not only met all requirements under TIERS but also produced 
information of high use potential; Furthermore, the results appeared to 
be of relevance to Chapter i projects implemented in other locations. 

PROGRAM CONTEXT 

The Hawaii State Department of Education is the only centralized, 
single state system in the nation. The state is called the "melting pot 
of the Pacific", its population being made up of people with varied 
ethnic, cultural and linguistic backgrounds. The Department of Education 
serves over 200,000 students across the state. A large percentage of 
this number are either immigrants or secbhd-gerierat ion children of 
non-English speaking parents. 

The Honolulu District is the largest of seven school districts in 
Hawaii. It serves a total of 55 public schools (40 elementary, nine 
intermediate arid six secondary schools), five special schools (e.g., the 
Hawaii School for the Deaf and Blind), 51 community schools for adults, 
55 private regular schools, six schools for special education, and 24 
private trade, vocational or technical schools ; The District serves a 
total of over 35,000 students every year. The Compensatory Education 
Section serves eligible disadvantaged students from both public and 
private schools. 

Since the implementation of the Comprehensive Language improvement 
Program (CLIP) in 1979, over 10,000 disadvantaged children have been 
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served in the district. These students were enrolled in some 24 public 
schools and 12 private affiliate schools covering grades two through 
twelve. 

The project schools were selected for having high concentrations of 
children from low income families. Students attending these schools came 
mainly from families receiving welfare assistance , living in public 
housing projects and/or high-density neighborhoods with a high 
concentration of immigrant or itinerant families. 

The students in the Chapter 1 program were selected on the basis of 
their test scores on the MAT reading comprehension test or the SAT 
reading test. The average cutoff score has been the 25th percentile or 
lower in some instances because of budgetary constraints. 

The etIP has four major components: language improvement centers, 
curriculum, instruction, and parent involvement. These components are 
integrated into a comprehensive program^ implemented and monitored by a 
highly-trained staff. 
Language improvement Centers 

All Chapter 1 activities are carried out in language improvement 
centers. These centers are located in project schools but are 
significantly different from a regular classroom in a number of ways. 
First, each center is staffed by a project teacher, one or more part-time 
teachers, one or more educational assistants, and a parent tutor. 
Secondly, the centers are structured in such a way that it is possible 
for several tutors to work with individual students or small groups of 
students simultaneously. Thirdly, a wide variety of preselected learning 
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materials are provided in the centers for use by individual students. 
The centers are set up to conform with the basic philosophy and 
principles of psycholinguistics. 

Curr iculam 

The curriculum implemented in the CLIP is based on the principles of 
psycholinguistics. The emphasis is on whole and natural language 
development. The whole language approach encourages the integration of 
all the language arts and concentrates on larger units, i.e. , larger than 
words or letters. Students are provided with many types of materials for 
abundant reading opportunities and resources emphasizing total language 
approaches , e.g. , assisted reading, uninterrupted sustained silent 
reading, and language experience methods. Additional curricular 
resources include materials which promote exper imenv *tion and 
problem- solving by students. Standard audio-visual eq 'ipment and 
hardware are made available to support the curricular emt^asis. These 
include tape recorders, language masters, record players, c«: -»eras, 
typewriters, and films trip projectors. 

ft curriculum plan is prepared by the Chapter 1 project teacher for 
each program participant in consultation with the class room/English 
teacher. The curriculum plan is based on the results of diagnosis and is 
designed around a cluster of specific learner objectives. It focuses on 
total language instead of isolated or fragmented reading skills and 
emphasizes a comprehension-centered language improvement approach. The 
diagnostic procedures include reading miscue analysis and the use of the 
Woodcock Reading Test anrJ the MAT Reading Instructional Tests. The 
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individual curriculum plan is periodically evaluated arid changes are made 
as needed. 

Instruction 

Instructional activities in the language improvement centers focus 
mainly on individual student needs. Instruction is guided by a set of 
performance objectives and clusters of instructional strategies. There 
is a strong interrelationship between the various performance objectives, 
instructional strategies and educational materials utilized. Low 
pupil-teacher ratio is maintained at all times through efficient 
scheduling and planning of activities. Inservice training of project 
teachers covers diagnostic procedures * the use of recommended 
instructional strategies as well as principles of psychblinguistics . 
Workshop topics include components of a comprehension-centered reading 
program/ language variations* and language-based problem-solving. 

Parent Involvement 

A unique and significant component of the CLiP is parent 
involvement. A district parent involvement assistant is responsible for 
coordinating and monitoring parent involvement efforts. A School Parent 
Advisory Committee (SPAC) is formed in each of the project schools to get 
parents involved in Chapter 1 projects. The SPACs also sponsor workshops 
in parenting skills and in helping Chapter 1 students read at home. A 
District Parent Advisory Council (DPAC) is established to coordinate and 
plan districtwide parent involvement activities. Its main purpose to 
provide parents the opportunity to become involved in program improvement 
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arid other activities that they deem helpful arid impbrtarit iri meeting the 
needs of the Chapter 1 students. The DPAC* whose membership consists of 
representatives from the various Chapter 1 school projects, acts as a 
liaison between parents arid their respective schools arid helps promote 
better school-home relationships. 

Specific program objectives of the CLIP include the following: 

1. The average normal-curve-equivalent (NCE) gain scores of 

CLIP students as measured by the Metropolitan Achievement Tests 
will be significantly higher than the no-treatment expectation. 

2. There will be an upward trend in the average NCE gain scores of 
CLIP students over the years. 

A majority of the CLIP students will make positive NCE gains. 
There will be an upward trend in the percentage of CLIP students 
making positive NCE gains; 

EVALUATION STRATEGY 
A sequence of key events led to the implementation of the 
improvement-or iented evaluat ion , includ ing : 

Evaluation focus. After having implemented TIERS for several 
years the CLIP project staff generally shared the perception 
that TIERS information alone was not sufficient for program 
implementation and improvement purposes; The evaluation focus 
needed to be broadened to include specific questions relating to 
staff concerns. 
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Formulation of evaluation qu e stion s . Several questions were 
subsequently formulated by the project staff to be addressed in 
the improvement-or iehted evaluation: 

(a) What are the effects of the CLIP on student achievement oh 
a short-term (annual) basis? 

(b) What parent activities, if any, contribute to student 
achievement? 

(c) How severe are the summer drop-off effects, if any, on 
student achievement? 

(dj long-term (multi-year) project participants continue to 

benefit from the program? 

(e) What instructional strategies are related to or produce 

high achievement? 
Selection of evaluatdrs . The district recognized that 
addressing these evaluation questions required more staff 
resources and expertise currently available within the 
district. An external evaluatbr was subsequently identified to 
design and implement the necessary evaluation tasks. A district 
staff was assigned half-time to provide assistance and to 
coordinate the evaluation activities. 
Planning and designing the evaluation , ft multi-year, 
improvement-oriented approach was used in the evaluation. WMile 
Model Al was used to collect achievement information, other 
data gathering strategies were developed to collect the 
pertinent data. 
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Evaluation implementation. Various data gathering instruments 
were developed by the external evaiuatbr. Most of the data 
collection tasks were carrica but by project or district staff 
under the supervision of the external evaiuatbr. Extensive use 
was made of Chapter 1 TAG services in completing the various 
evaluation tasks. 
Three major types of evaluation activity were conducted during the 
1979-82 school years. These included the annual external evaluations, 
the sustained effects evaluation and the longitudinal data base study. 
These activities were an integral part of the CLIP and served both a 
formative (for improvement) and a suiranative (for accountability) function 
(Nevo, 1983) . 



EVALUATION STUDIES AND RESULTS 

Program Impact 

For each of the three school years in 1979-82 an annual evaluation of 
impact was conducted on the CLIP by the external evaluator. The SMS 
Research Corporation, a Honolulu based firm> used a prescribed set of 
procedures to conduct the evaluations. The procedures, approved and 
promulgated by the U.S. Department of Education, are fully described by 
Tallmadge et al. (1981). The following is a brief description of the 
rationale and essential elements of the process used in the annual 
evaluations . 
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Procedure ; Three "models n are included in the TIERS for the conduct 
of Chapter 1 evaluation; The models provide an observed measure of 
performance at the end of the project and ah estimate of what that 
performance would have been without the project. The estimate of how 
well students would have performed without the project is called the 
no-treatment expectation. The difference between the observed 
post-project performance and the no-treatment expectation is taken as a 
measure of project impact. In other words, the models answer the 
question: How did the project change student achievement scores from 
what they would have been without the project? 



Project Impact = 




Gains resulting from Chapter 1 projects are reported in normal curve 
equivalents (NCEs) . The NCE scale is tied to the distribution of test 
scores of a nationally representative sample of students arid matches the 
percentile ranks of that distribution at values of 1, 50, and 99. When 
nationally normed tests are used, NCEs can be derived directly from the 
percentile rank cor respond ing to each possible test score. 

Of the three TIERS models, the most widely used is the 
norm-referenced model or Model A (Stonehill and Anderson, 1982) . The 
Honolulu District adopted the model in 1977 and has since been using it 
to evaluate all Chapter 1 projects in the district. In Model A, 
normative data are used to substitute for data from a comparison group. 
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The no-treatmerit expectation is derived under the assumption that 
students will maintain their status with respect to the norm group p rdm 
pretest to pbsttect (Wood, 1980) • In other words, it is assumed that 
students who scored at the 20th percentile on the pretest would also 
score at the 20th percentile on the pdsttest Unless they participate in 
some special instructional activities. If they participated in such an 
intervention and it had a positive effect, the project students would 
achieve a higher percentile status on the posttest than they did on the 
pretest. For the norm-referenced model , project impact is defined as 
shown below. The first term in the equation 

corresponds to the "observed post-project performance" and the second 
term, to the "nb-treatmeht expectation." 



It should be emphasized that Model ft does not compare the achievement 
gains of Chapter 1 students with the gains of "average" students. 
Instead, the gains of project students are compared against the gains 
expected of norm-group students who scored at the same percentile level 
on the pretest. 

Obviously, the validity of Model ft hinges on the similarity of 
project students to students in the norm group with whom they are 
compared. Thus it is essential that all comparisons be made with 
normative data compiled from students at the same age or grade level as 
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the project students. For example^ if a faii-to-spr ing evaluation is 

conducted on a Chapter 1 project serving fifth-grade students, pretest 

status is determined by beginning-of-f if th-grade norms and posttest 

status is determined by end-of-f if th-grade norms. 

Testing , ft proper implementation of Model ft requires that all 

testing be accomplished at* or near> the midpoint of the time interval 
during which the national norm group was actually tested. For all three 

school years included in the study, the Metropolitan Achievement Test, 
1978 was used to pretest and posttest all CLIP students in the district. 
The reading comprehension score was chosen as a measure of program impact. 
Functional level testing was used to ensure that the test level would 

not be too easy or too difficult for the project students. In general* 
students were tested one level below their grade. Students were tested 
by project staff in October (pretest) and in April (posttest). Except 
for a few isolated cases, a majority of the project students were pre- 
arid posttested. There was rib reason to suspect that the small number of 
students riot pre- arid posttested presented a potential bias in the 
results. 

Results . For each of the three school years, achievement gains were 
calculated for each grade level and in clusters of grade levels (i.e., 
elementary and secondary). These are presented in Tables 1 through 3. 
Overall, NCE gains ranged from 5.1 to 6.0. Consistent with the national 
trend (Stonehill and Anderson, 1982) the elementary grades made higher 
gains than the secondary grades. This is perhaps more a result of the 
developmental process than a consequence of differential effectiveness of 
the program. 
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Table i 



Chapter 1 Reading Achievement Data 
Honolulu District (1979-80) 



Grade Level 


O LUU CllUo 

Tested 


Average NCE Scores 
Pre Post 

Test Test Gain 


Percent 
of 

Students 
Who Gained 


9 


** X / 


22.3 


26.2 


3.9 




55 


*5 


306 


14.9 


31.5 


16.6 




86 


A 
H 


O £, y 


25.0 


32.4 


7.4 




77 


5 


333 


22.8 


29 .4 


6.4 




73 


6 


224 


21.7 


31.3 


9.6 




88 


7 


466 


27.9 


31.3 


3.4 




61 


8 


451 


26.6 


29.9 


3.3 




59 


9 


431 


23.7 


25.1 


1.4 




51 


10 


275 


15.2 


18.6 


3.4 




54 


11 


268 


14.6 


15.8 


1.2 




43 


12 


192 


9.4 


12.8 


3.4 




53 


TOTAL 


3,692 


21.6 


26.7 


5.1 




63 




Note: No standard 


deviations 


were obtained 


for each 


grade for 


the 





1979-80 data. 
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Table 2 



Chapter 1 Reading Achievement Data 
Honolulu District (1980-81) 



Students Average NCE Scores Percent 

Grade Level Tested Pre Post Gain Std. of 



Test Test Dev; Students 

Who Gained 



2 276 17.1 28.0 10.9 18.6 67 

3 318 18.5 33.1 15.6 12.8 84 

4 310 26.6 33.7 7.1 10.2 75 

5 328 26.1 30.4 4.3 8.9 65 

6 323 26.4 33.5 7.1 7.7 81 

7 349 27.6 31.2 3.6 10.0 63 

8 450 26.4 29.0 2.6 9.5 56 

9 459 23.5 26.2 2.7 10.2 55 

10 317 18;4 22;6 4.2 9.7 62 

11 292 19.0 2i;7 2;7 11.6 48 

12 192 14.5 19; 4 4.9 11.5 52 

TOTAL 3,610 22.7 28.4 5.7 4.7 64 



Std. Dev. = Standard deviations of gains scores. 



Table 3 



Chapter 1 Reading Achievement Data 
Honolulu District (1981-82) 



Students Av e rag e NCE Scores Percent 

Grade Level Tested Pre Post Gain Std. of 



Test Test Dev. Students 

Who Gained 



3 251 20.6 36.2 15.6 12.8 87 

4 216 24.2 31.4 7.2 10.4 72 
; 5 282 25.5 32.3 6.8 9.7 74 

6 238 25.6 34.1 8.5 7.7 87 

7 366 27.9 33.0 5.1 8.6 71 

8 406 26,2 29.3 3.1 8.9 58 

9 396 23.7 25.7 2.0 10.2 49 
id 154 17.6 24.7 7.1 10.1 69 

11 89 16.6 20.0 3.4 7.5 53 

12 98 13.3 16.8 3.5 9.6 57 

TOTAL 2,496 23.8 29.8 6.0 10.4 68 



Std. Dev. = Standard deviations of gain scores. 
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Table 4 



Longitudinal Analysis of Chapter 1 Reading Achievement Gains 
Honolulu District (1979-82) 











Gain 


Percent of Students 


Projects 




Students 






with Positive 


School Year 


Tested 


Mean 


SD 


Gain Scores 


Elementary 


1979-80 


1,637 


8.3 


12.7 


87 


School 


1980-81 


1,552 


8.9 


12.6 


75 


Projects 


1981-82 


956 


9.6 


10.6 


80 


Secondary 


1979-80 


2,055 


2.7 


10.1 


54 


School 


1980-81 


2,067 


3.2 


10.3 


56 


Projects 


1981-82 


1,465 


3.6 


9.3 


60 


Note. All 


NCE gains were 


statistically 


significant at the 


.01 level. 
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The percent of project students making positive NCE gains provides a 
separate measure of project impact. The data showed that in most 
instances a predominant majority of the project students made positive 
NCE gains. 

Table 4 presents a longitudinal perspective of achievement result? 
from 1979 to 1982. The data are presented for elementary and secondary 
projects separately. The information includes number of students 
pretested and pbsttested, average NCE gain scores* standard deviations of 
average gain scores, and the percentage of students making achievement 
gains. 

At the elementary level, both the average gain and the percentage of 
students making gains increased consistently over the three years, ft 
similar trend was observed for the secondary schools. These achievement 
patterns are particularly significant in that they suggest cumulative 
impact of the CLIP on student achievement. 

Parent Involvement 

The involvement of parents of CLIP students in program activities is 
an integral element of program design. In Hawaii , parent involvement has 
long been an essential component of Chapter 1 operations. In Honolulu 
district, parent involvement differs from similar efforts in other 
districts in that its delivery and coordination of parent training is 
closely articulated with the psychol ingu is t ic classroom strategies. The 
parent involvement study, conducted in the 1980^81 school year, dealt 
with the effects of parent training on student achievement. The 
following discussion focuses on parent behaviors which have been found to 
be related to student achievement gains. 
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As shown in Tablb' 5, students whose parents helped them with their 
homework made greater NCE gains than those whose parents did hot provide 
such help. Similar differences were found between students whose parents 
encouraged them to read and those who received no encouragement from 
their parents. For items more central to parent behaviors fostered by 
the CLIP (e.g.* visiting the library, asking Chapter 1 teachers for 
assistance, setting a family reading hour), differences in gain scores, 
while hot dramatic* were always in the expected direction. Parents who 
reported these behaviors had children who tended to make higher gains. 

Chapter 1 makes the implicit assumption that a combination of 
remedial instruction at school and full parental involvement in the 
educational process would raise the achievement level of disadvantaged 
students. While there was ample evidence to demonstrate the impact of 
supplemental instruction, the present study provided the first indication 
in Hawaii that appropriate parent involvement is associated with 
increased reading achievement. 
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fable 5 

Parent involvement and Student Reading Achievement 
Honolulu School District (i980-81) 



Survey Items and Parent Responses Respons es 

No. Gain 



Do you do any of the following things for your 







143 


6.7 




(N) 


8 


3.1 




(Y) 


137 


6.9 




(N) 


15 


3.2 


Have discussions about materials child 


(Y) 


127 


7.2 




. (N) 


23 


3.2 




m 


127 


6.8 




(N) 


26 


5.2 


Set a specific time and provide a quiet 


(Y) 


104 


6.1 


place for your child to study each day . < 


- (N) 


44 


7.0 




(Y) 


79 


6.7 


(N) 


71 


6.2 


Ask the Title! teacher how you can 


(Y) 


77 


6.9 




i (N) 


69 


5.8 


Set a specific time for the whole family 


(Y) 


45 


7.2 




(N) 


104 


6.4 


Teach child how to follow directions . i , 


. (Y) 


122 


5.5 




(N) 


27 


10.6 


Talk about TV programs you watch 


(Y) 


134 


7.0 




(N) 


16 


2.7 


Discuss current events with your child . . 


. (Yj 


120 


6.0 




(N) 


29 


9.7 


Take child on outings as a learning 


(Y) 


94 


9.0 




(N) 


9 


4.0 


Parents Responding 




153 


6.5 



Y = Yes 
N _■ No 

No = Number of parents responding Yes or No. 

Sain = NCE gains of students whose parents responded Yes or No to the 
survey items. 
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Summer Drop-Off 

Over the past decade, evaluation results indicated that Chapter 1 
activities have been effective in raising the achievement level of 
participating students* Some researchers (e.g., David and Pelaviri, 1978) 
pointed but, however, that the achievement gains were hot sustained over 
the summer months. They suggested that large achievement gains produced 
during the school year were offset by comparable losses during the 
summer. The hew Chapter 1 legislation addresses the issue of sustained 
effects by requiring the evaluation of educational achievement over a 
twelve-month (or longer) period. This is to determine whether gains made 
during the regular school year are sustained over the summer. 

A sustained effects evaluation was conducted in the Honolulu district 
during the 1981-82 school year. Data were obtained from files containing 
student pretest and posttest scores from the 1980-82 CLIP program 
evaluations. Files were matched oh a student-by-student basis across the 
school district. This made it possible to include in the analysis 
students who were in the program for two consecutive years, but at 
different schools. 

Districtwide results of the sustained effects analysis for students 
enrolled in the CLIP during the 1980-81 and 1981-82 school years are 
presented in Table 6. Summer effects between spring of 1981 and fall of 
the same year were negative. It was evident that CLIP students did 
experience some reading achievement loss during the summer. The negative 
summer effects were statistically significant at the .05 level. On the 
other hand, the 12-month gains of the students were positive and also 
statistically significant at the .05 level. It thus appeared that while 
summer regression aid exist, the loss was hot of sufficient size to 
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offset the 9-month gains; The last column in Table 6 shows the 1981-82 
gain scores for students who were also enrolled iri the 19^0-81 program. 
The data suggest that the 9-month gains of returning students exceeded 
their gains in 1T80-81. 



Table 6 

Sustained Effects Achievement Data 
Honolulu District (1981-82) 



N 1 Fall Spring Fall Spring Fall to Fall to 1981 Fall 

Spring Fall Summer Sprin 

1980 1981 1981 1982 1980-81 198' -81 Effect 1981- 

1,301 21.5 26.2 25.0 30.2 O O -1.2 5.2 

^"Number of students with all required test scores, .Includes only those students 
were enrolled iri the CLIP in both 1980-81 arid 1981-82 school years. 



NCE Test Scores NCE Achievement Gain 



Longitudinal Effect s 

The Honolulu district conducted a longitudinal study to track CLIP 
students for school years 1979 through 1982. The study was prompted by a 
general concern over the effects of long-term participation in the 
program. It has been suggested that long-term participants form a core 
group of disadvantaged youngsters who for one reason or another do not 
benefit from the program; In implementing the bcudy^ a data file was 
created to contain scores of students who had been in the program for 
three years. Some 684 such students were identified in the district. 
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As depicted in Figure 1, the three-year participants entered the CLIP 
in 1979 with ah average pretest score of 19.5 NCEs. In the first year } 
they gained 4.9 NCEs followed by a summer loss of 2.0 NCE points. A 
second year of remedial instruction, however , produced an additional gain 
of 4.4 NCEs, again, followed by a loss of 1.6 NCEs in the summe-. 
For the 1981-82 school year, these students gained an additional 4.5 
NeEs, making an overall gain of 10.2 NCEs. 

The hypothesis that these students represent a residual group who do 
not benefit from program services is not supported by the data. Instead, 
the 684 students (about 19 percent of the 1979-80 CLIP population in 
Honolulu) consistently made achievement gains, even though their gains 
may not be as high as those made by other CtlP participants. 

Instructional Strategies 

Much has been said about the conduct of evaluations which result in 
little or no use of evaluation information for program planning and 
improvement (Wise, 1978; Thompson arid King, 1981). Most state 
departments of education arid local school districts have more evaluation 
information at their disposal thari they could meaningfully examine arid 
use in making program decisions. On the other hand, tiost evaluations are 
performed in a fashion that is riot conducive to evaluation use. They are 
often narrow in scope, having been designed to meet specific federal or 
state requirements. As a result, little or rid relevant information is 
provided in a timely fashion for making program decisions. The CLIP 
instructional strategies study represents a conscious effort on the part 
of the project staff to promote evaluation use in the district; 
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Figure 1: Reading Achievement of Students 
Enrolled in the CLIP During School Years 1979-82 (N*684) 
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Exploratory discussions on the study began in late 1978 ; These 
discussions stemmed from project staff's concern over the availability of 
a great amount of data kept in the district files and the lack of use of 
these data for planning and improving Chapter i activities in the project 
schools. A series of consultations were held between TAG and project 
staff during the latter part of 1978 and early 1979. By June 1979, 
variables of interest were identified and specifications for coding the 
variables were developed. 

The instructional strategies study was conducted in early 1980 on 
data obtained during the 1978-79 school year. The study included four 
schools (Jefferson, Central, Kaiulahi and Puuhale) With a total of 383 
students in grades 2-9. Student achievement was measured by the Woodcock 
Reading Mastery Test (1973). The data on identified variables were 
gathered from district data files maintained for Chapter 1 evaluation 
purposes. Data coding Was performed by the Honolulu district staff. 
Completed data sheets were forwarded to the TAe for key-punching and 
analysis. Descriptive and inferential statistics were obtained from the 
data to illustrate the range of copies or evaluation questions which 
might be addressed by the data. For example , correlations between 
instructional strategies and student achievement gains were obtained. 

Several instructional strategies were found to be related to student 
outcomes. These strategies included: read aloud/read along/ assisted 
reading techniques, reference skills, questioning strategies, extended 
use of teacher/pupil made materials, book reports, language-thinking 
approaches, selected games/activities for manipulating language, and 
reading strategy lessons. 
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Table 7 



Correlations Between Instructional Strategies and 
Achievement Gains in Passage Comprehension 
(1978-79 School Year) 



Grade Levels 

Strategies Second Third Fourth Fifth Sixth Seventh Eighth Ninth 



_ (N=66) (N=47) (N=33) (N=37) (N*53) (N=43) (N*40) (N=37j 



1. 


Read aloud/ 
Read along 


• 39 






.28 




2. 


Assisted 

Reading 

Techniques 








.36 




3. 


Uninterrupted 
Sustained 
Silent Reading 












4. 


Entire Range 
of Reading 
Materials 








.37 




5. 


Math, Science 
Social Studies, 
Literature 










.32 


6. 


Dictionary, 
Library 

Other Reference 
Skills 










.32 


7. 


Pars, Manzo' s 
Technique, etc. 


.58 










8. 


Questioning 
Strategies 


.35 










9. 


Dialogue/ 
Discussion 












10, 


Written 
Conversation 












li. 


Extended Use 
of teacher/ 
Pupil-Made 
Mater ials 


.47 




.47 


.36 




12. 


Book Reports 








.42 
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Table 7 (Continued) 



Grade Levels 

Strategies Second Third Fourth Fifth Sixth Seventh Eighth Ninth 
(N=66) (N=47) (N=33) (N=37) (N=43) (N=43) (N=40j (N=37) 



13. Constructions/ 

Products .31 

14. Reading Records 

15. Language- 
Thinking 

Approaches .41 .49 .38 

16. Written 
Conversation 
(Watson) 

17. Materials That 
Present 
Distinctive 
Language Patterns 

18. Materials 
Involving 
Listening/ 
Wr it ing 

or Combination 
of Both; 
Storytelling, 
Reading Aloud, 
etc. 



19. Sequenced 
Combinations 
of Visual, 
Audio and 
Reading 
Materials ; 
Thematic Units 
and Projects 

20. Selected Games/ 
Activities for 
Manipulating 
Language 



,30 



.31 
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Table 7 (Continued) 



Grade Levels 

Strategies Second Third Fourth Fifth Sixth Seventh Eighth Ninth_ 
{N^j&J 1^31) -1^53} (N=^3j {N^iO) (N=37) 



21. RMI, Short- 
Form 

22. Informal 

Applications .50 .31 

23. Reading 
Strategy 

Lessons .50 



Note: Only significant (p< .05) correlations are reported. 

Correlations were hot computed for strategies used by all or 
none of the students. 
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QUALITY MONITORING 

The Process 

Quality monitoring is a participatory and collaborative effort to 
bring about positive changes in project schools. it is a systematic 
process of examining project implementation variables and evaluation 
results and translating them into action plans to improve student 
achievement. A dynamic process, quality monitoring promotes school level 
use of evaluation data for program improvement, leads to an on-going 
identification and analysis of variables affecting student achievement 
and ensures fidelity of program implementation. An important consequence 
of quality monitoring is the continuous exploration for better ways of 
providing instruction to Chapter 1 participants. The identification and 
analysis of important variables provide a sound basis for examining the 
organization, structure and effectiveness of the etIP. 

Quality monitoring begins at the district level with the district 
staff assuming the following responsibilities: 

- Examination of achievement gains for each project school in 
comparison to previous years, to district averages , and to grade 
level averages. Significant patterns are noted. 

- Examination and analysis of project implementation variables. 

- Preparation of project personnel/administrator for the Initial 
quality monitoring meeting by providing a list of topics to be 
discussed . 
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- Coordination of meetings with school project staff arid 

administrators to identify variables/practices in the areas of 
administration , program implementation , monitor irig, testing arid 
evaluation, curriculum arid inservice , arid parent involvement. 
As program implementation arid outcome variables are reviewed, 
questions such as the following are raised: 

1. Does the staff irig configuration produce a classroom environment 
conducive to maximal language learning? 

2. Is the instructional delivery system meeting the needs of 
project participants? 

3. Is the per pupil cost noticeably higher or lower than others? 
Why? 

4. Are the evaluation procedures appropriate for the Chapter 1 
population? 

5. Are the evaluation results significantly better or worse than 
those of previous years? 

6. Do the evaluation results indicate significant patterns this 
year, from year to year , and/or over the years? 

7. What are the strong and weak areas in the test performance of 
project participants? 

8. Are project staff using appropriate instructional strategies and 
materials? 

9. What are the iriservice needs of project staff? 

10. Are parent involvement activities helping parents to help their 
children at home? 
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Then in a collaborative, problem solving mode, district and school 
level project personnel and administrators develop a plan of action and 
identify persons responsible for carrying out the plan; Decision making 
and responsibility are shared by ail individuals involved in the project. 

The following chart describes the quality monitoring process: 



QUALITY MONITORING PROCESS 





District 






Staff 










Evaluation of 




1 — ► 


Student 


4 




Outcomes 






1 


— » 


Analysis of 






Variables 






1 


— + 


Action Plan 












Monitoring 






- School 






- District 






— i 




Refinement 






Of CLIP 






4 




Student 




- 


Achievement 





Since the quality monitoring process was first implemented in 1980, 
project schools selected to participate in quality monitoring have met 
with much success in using evaluation results for program improvement. 
Achievement gains have invariably resulted from the total commitment of 
the project personnel to this objective; In addition, several other 
positive outcomes were noted when the participating schools began the 
second year of quality monitoring: 

1. The district's role in examining evaluation results diminished 
as school project personnel began to analyze evaluation results 
themselves. 

2. Project personnel began to participate in program improvement 
plans with a positive attitude focusing on solutions rather than 
problems , 

3. ft self monitoring attitude prevailed. School project personnel 
were better able to identify problem areas; This was evident in 
quality monitoring sessions where school project personnel had 
determined the priorities for the year and had identified 
problem areas to be discussed. 

Thus, the quality monitoring process has encouraged problem solving 
in ah objective and systematic manner. With this new perception, 
positive changes have continued to occur in the participating schools. 

Ah Example 

During the 1982-83 school year , quality monitoring was implemented at 
Dole Intermediate School. The third largest intermediate school in the 
district, Dole had an enrollment of 1,096 in grades seven through nine. 



30 



doc 0373t 




Ethnic backgrounds of the student population included Filipino (45 
percent) > Samoan (24 percent), part Hawaiian (12 percent) and Japanese (5 
percent) . Forty-three percent of the students were in the free/reduced 
lunch program. 

The Chapter 1 project served some 400 youngsters in grades seven 
through nine. The project employed 15 staff members including 5 
teachers, 6 part-time temporary teachers, 3 educations 1 assistants and 1 
para-professional tutor. Total project funding in 1982-83 was $253,793 
with a per pupil cost of approximately $634. 

In examining the evaluation results for the 1980-81 and 1981-82 
school years, district staff noted that Dole Intermediate School's 
Chapter 1 classes in grades eight and nine had not made the anticipated 
gains. The ninth graders in particular showed negative NCE gains for 
both school years. The decreasing gains prompted the selection of the 
school for quality monitoring. Major events which occurred during the 
process are described as follows: 

Quality monitoring meeting . A standard initial school visit form was 
sent to the school project administrator and staff prior to the scheduled 
meeting. Thr initial schcxsl visit occurred at the beginning of the new 
school year in October. During the quality monitoring meeting, attention 
focused on the poor achievement of the Chapter 1 participants in grades 
eight and nine. Variables thought to have some impact upon the low 
achievement trend were identified . Accordingly, the quality monitoring 
group developed an action plan to effect change, identified persons 
responsible for various tasks, and set up timelines for carrying out the 
plan. Specifically, it was agreed that the grouping assignments iu 
grades eight and nine did not permit small group instruction or other 
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optimum instructional arrangements. The plan of action was to reassign a 
Chapter 1 project teacher from grade seven to grade eight. Project 
students were reassigned so that the larger Chapter 1 classes were 
reorganized into smaller classes. The school administrator , the district 
resource teacher, the school registrar and the project teacher 
coordinator were all responsible for the reorganization which took place 
shortly after the quality monitor ing meeting. 

Implementation of action plan . Immediately following the initial 
quality monitoring meeting, the district resource teacher met with the 
project staff to review the plan of action and to work out farther 
details. Three specific areas needing attention were identified to be 
delivery of services, staffing and curriculum, ft detailed plan of action 
to effect changes in these areas is presented in Table 8; 
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Table 8 



Program Improvement Plan for Dole Intermediate School 
1982-83 School Year 



Delivery of Services Staffing Curriculum 



GRADES 8 & 9 

- Mostly large group 
instruct ion 

- Students assigned 
PROBLEM to Chapter 1 

classes by grade 
levels, hot heeds. 
Some classes 
overloaded 



SCHEDULES 

PTTs assigned to 
teachers and 
utilized as 
aidjs rather 
than tutors. 
School has 
rotating schedule . 
PTTs ' schedules 
do not correspond 
to pupils' 
schedules 



TESTING 

- Lack of test taking 
skills 

- Poor attitude 
toward test taking 

- Oyer testing with 
MAT 

GROUPING 

- Shift from large 
group instruction 
to small group 
requires curricular 
modifications 



\CTION 



Reassign Chapter 1 
project teacher in 
grade 7 to grade 8 
Make lateral 
tranfers from 
large Chapter 1 
classes to smaller 
classes 



- Identify PTT 
roles and 
responsibilities 
through class- 
room observation/ 
interview 

- Meet with 
project staff/ 
administrator 

to report results 
of observations 
and interviews 

- Idertify strengths 
and weaknesses and 
formulate recommen- 
dations 



District to provide 
"mini tests" for project 
staff to teach test 
taking skills 

Coordinate with district 
SLEP 

Administer reading 
diagnostic tests 
Review/revise 
personalized educational 
plan on student record 
profile 
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Table 8 (Continued) 



Program Improvement Plan for Dole Intermediate School 
1982-83 School Year 



Delivery of Services Staffing Curriculum 



- Administrator , 
district resource 
teacher , 
registrar , 

PERSON project teacher 

RESPONSIBLE 



- District 

resource teacher, 
project staff, 
administrator 



" ? r 93? c *Lstaf f , 
district staff 
project teachers , 
district resource 
teacher 



TIMELINE 



- As soon as 
possible by 
pretest date 



As soph as 
possible 

Upon completion 
of observations 



- All Year 



P re-post testing 
periods 



- All year 









- More equitable 


- Meeting held 


- Project teachers 


class size 


- PTTs assigned to 


reported students 




specific students 


appeared more 




- Coordinated effort 


confident and motivated 




evident in planning 


during posttestihg 


RESULTS 


- Successful 


- SLEP used Chapter 1 




practices of 


reading test scores 




project shared with 






entire project 


- Project teacher more 




staff 


aware of wide range of 






needs 






- Purvis functioned at 






own levels 
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As other concerns/problems arose during the project years, new plans 
were developed. Some of the changes occurred immediately and others were 
made later during project year. Outcomes reported by school project 
staff were reviewed by district staff at the beginning of the following 
school year as a new quality monitoring cycle began. 

The quality monitoring process required total involvement of all 
project personnel. On an on-going basis, the project staff shared 
information, insights and evaluations with the district resource teacher 
and other colleagues. The involvement of the district resource teacher 
as a team member provided the opportunity to determine what was really 
occurring in the classrooms. 

Outcomes . Posttests were administered in April 1983. Results 
obtained by means of the A-l Norm Referenced Model are shown in Table 9 
and Figure 2. Significant gains were made by students in grades eight 
and nine. The project staff accomplished their goal of raising the 
reading achievement levels of the eighth and ninth graders at Dole 
Intermediate School. 
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Table 9 



Chapter 1 Reading Achievement Data 
Dole Intermediate School (1981-83) 



Grade Level 



Students 
Tested 



Average NCE Scores 
Pre Post Gain 

Test Test 



Percent 

Of 
Students 
Who Gained 



7 
8 
9 



113 
( 98) 

106 
(117) 
96 
(123) 



28.7 
(31.5) 

27.8 
(27.9) 

24; 9 
(27;5) 



34;l 
(34.8) 

31.0 
(35;l) 

23;2 
(32.6) 



5;4 
(3.3) 

3.2 
(7.2) 

-i;7 
(5;l) 



80 
(63) 

59 
(78) 

36 
(73) 



TOTAL 



315 
;338) 



27,2 
(28.9) 



29.7 
(34.1) 



2.5 
(5.2) 



60 
(72) 



NOt€ 



Data for the 1982-83 school year are in parentheses. 
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10 
9 
8 

7 

6 

5 

4 

3 

2 

1 

6 
-1 

-2 

-3 



5.4 



111 



3.2 




Pip 



2.5 



Grade 7 



Grade 8 



Grade 9 



SCHOOL 
AVERAGE 



□ 



1981-82 



1982-83 



Figure 2: Dole Intermediate School NCE Gains 
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CONCLUDING REMARKS 



In the Honolulu improvement-oriented evaluation the evaluators were 
able to provide answers to pertinent evaluation questions. For example, 
for grades 2-12 the short-term effects of the Title I/Chapter 1 projects 
were indicated by NCE gains ranging from 5.1 to 6.0 for the 1979-82 
school years. On a longitudinal basis, cumulative impact was 
demonstrated for the elementary and secondary projects. Over the 
three-year period, NCE gains for elementary projects increased from 8.3 
to 9.6. Corresponding increases for secondary projects were from 2.7 to 
3.6 

ft number of parent activities were shown to be related to students 1 
NCE gains. These included: 

- Encourage your child to read 

- Help your child with homework 

- Have discussions about materials child has read 

- Listen to your child read 

- Set a specific time for the whole family to read 

- Talk about TV programs you watch together 

- Take child on outings as a learning experience 

The summer effect data suggested that there indeed was a summer 
drop-off. Based on a sample of 1,300 students, the decline was, however , 
shown to be very slight (-1.2 NCEs) . The multi-year project participants 
were able to continue to benefit from project services from year to 
year. Over a three-year period (1979-82) they were shown to have made an 
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overall gain of 10.2 NCEs . The hypothesis that these students represent 
a residual group who do hot benefit from program services is hot 
supported. 

Several instructional strategies were found to be related to student 
outcomes. These strategies included: read aloud/read along, assisted 
reading techniques, reference skills, questioning strategies, extended 
use of teacher/pupil made materials, book reports, language-thinking 
approaches^ selected games/activities for manipulating language, and 
reading strategy lessons. 

Results of the studies were presented to teachers^ teacher aides> 
parents and school administrators at the annual Chapter 1 program 
implementation conferences or through individual consultations with 
school project staff and principals at the various school sites. To 
promote a sense of involvement and ownership in the evaluation, project 
staff were asked to provide evaluation questions which the district 
should address. On an on-going basis possible responses to the questions 
along with supporting data were then shared with the staff. 

Through a quality monitoring process, district staff were able to use 
the evaluation data in working with school level staff to effect positive 
changes in program implementation. Outcomes of these program improvement 
activities as reflected in NCE gains were quite dramatic in some 
schools. Moreover, school level staff are gradually building their own 
capacity for using evaluation information to effect program improvement. 
A snowballing effect is becoming more and more evident. 

Outcomes of the Honolulu effort suggest that given the motivation and 
sufficient resources, a medium-sized local district could carry out ah 
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imprbvenient-br iehted evaluation to complement the implementation of 
TIERS • Information obtained from such an evaluation effort not only is 
of high potential for use but also could be of general relevance to other 
Chapter 1 projects in the state and across the nation. Such evaluations 
represent a tremendous resource for improving Chapter 1 services to 
disadvantaged youngsters. 
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